National Grid ESO are required by our License and by the REMIT regulation (EU Regulation on wholesale Energy Market Integrity and Transparency) to monitor the market for suspicious activity relating to manipulation, insider trading, breach of Grid Code etc. Our current, manual, processes are not infinitely scalable or transferable as the market grows so greater automation and sophistication is required.
The development of a more sophisticated, Machine Learning (ML) based solution will be investigated to increase the efficiency of team activities and be scalable to new products and increasing market participant numbers.
Benefits
Incorporating more datasets, including the interaction between different marketplaces, detection of cross market mechanisms to manipulate ENCC (Electricity National Control Centre) decision making will be more readily identifiable. This may encourage more timely notification of changes in operating profiles and prices to the ENCC, making the plan more secure and reducing decision making pressure.
Furthermore, by applying machine learning techniques, anomaly detection can be individualised to the resource economics, size, and technology types, enabling market monitoring to identify anomalies across new technology types, and better support all market participants in improving compliance with market rules, without unintentional bias to larger BM Units that may result from standard rules-based alerting. This will become more important as the energy system has greater participation from small energy providers in the energy transition.
Learnings
Outcomes
The successful outcome of the project will be a new suite of tools which allow for anomalies to be identified and investigated by the market monitoring team against the different REMIT principles. The tools are integrated into market monitoring processes and are understood and utilised within the team. At the end of WP2-WP4, a report describing the methods for finding and characterising anomalies for the type of manipulation alongside a proof-of-concept code for extracting the anomalies with visualisations will be produced. In the instance where the behaviour can’t be detected with the required confidence, the difficulties and the possible routes to improve the data will be identified.
Following the completion of the exploratory analysis for WP1, three written Jupyter notebooks were produced covering initial statistical analysis on the datasets with visualisations, initial pricing analysis in preparation for the commencement of WP2 and a comparison of the datasets between the National Economic Database and public datasets. These datasets include outturn prices across BMUs as well as forward price submissions which are confidential to the ESO, bid offer acceptances with the price and volume instructed, all dynamic data units submit, the de-rated margin and loss of load probability, intraday and day ahead market prices, temperature data and gas prices.
Lessons Learnt
The conclusion to WP1 determined that some internal datasets would enhance the capability of the tools for monitoring the types of manipulation and there were fundamental differences in naming conventions between the ESO and public datasets selected. It was considered not beneficial to continue conducting analysis on all these datasets given the volume of data provided in WP1 and especially where the data provides a similar level of information. Therefore, it was concluded to focus on a subset of data from the ESO database with some public datasets for the development of the models in the future work packages.
WP2 will set the standard of how the tools will be created and integrated into ESO’s systems and market monitoring processes for the future work packs and thus the outcome will be critical in setting the direction of the solutions for the rest of the project.
